refactor(spark): resolve format options through a VortexOptions type - #9191
Open
jackylee-ch wants to merge 1 commit into
Open
refactor(spark): resolve format options through a VortexOptions type#9191jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
jackylee-ch
force-pushed
the
spark-vortex-options
branch
from
August 5, 2026 16:51
4894175 to
0f69e66
Compare
Merging this PR will regress 1 benchmark
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Spark lower-cases the keys of the CaseInsensitiveStringMap it hands to a table, so an option a user spelled vortex.workerThreads arrived as vortex.workerthreads. Every call site that read an option had to know that, and none did: the worker-thread count was looked up case-sensitively and always fell back to its default, and VortexTable merged scan options over table options with a plain putAll, leaving the same option present twice under two spellings. Introduce VortexOptions, following the shape of Spark's own ParquetOptions: it is Serializable, holds its CaseInsensitiveStringMap as transient and rebuilds it on demand, and gives every option one place that names it, documents its default and validates it. withOverrides matches keys case-insensitively so an override replaces the option it means to. Thread it through the read and write paths in place of the raw map, converting back with asMap() only where the native bindings are called, and drop the camelCase seeding VortexScanBuilder used to compensate. VortexSparkSession.get and the writer's batch-size resolution now go through it too, so vortex.session.provider and vortex.write.batch.size resolve case-insensitively as well. Signed-off-by: jackylee <qcsd2011@gmail.com>
jackylee-ch
force-pushed
the
spark-vortex-options
branch
from
August 6, 2026 02:29
0f69e66 to
210dac6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the discussion on #9170: rather than fixing the case-insensitive lookup at one call site, this gives the Vortex format options a type.
Why
Spark lower-cases the keys of the
CaseInsensitiveStringMapit hands to a table, so an option a user spelledvortex.workerThreadsarrives asvortex.workerthreads. Every call site that read an option had to know that, and none did:VortexPartitionReaderFactorylooked the worker-thread count up case-sensitively, so any spelling other than the exact camelCase one silently fell back to the default.VortexTable.newScanBuildermerged scan options over table options with a plainputAll, so the same option could end up present twice under two spellings, andVortexScanBuilderseeded the camelCase spelling to compensate.vortex.session.providerandvortex.write.batch.sizehad the same exposure, each parsed inline with a string literal.What
VortexOptionsfollows the shape of Spark's ownParquetOptions:Serializable, holding itsCaseInsensitiveStringMapastransientand rebuilding it on demand, so it crosses the serialization boundary intoVortexFilePartitionand the reader/writer factories without carrying a non-serializable field. Every option now has one place that names it, documents its default and validates it:WORKER_THREADSvortex.workerThreadsworkerThreads()— default 4, rejects a negative countWRITE_BATCH_SIZEvortex.write.batch.sizewriteBatchSize()— default 2048, falls back outside [1, 65536]LEGACY_WRITE_BATCH_SIZEbatch.sizeSESSION_PROVIDERvortex.session.providersessionProvider()withOverridesfolds keys withtoLowerCase(Locale.ROOT)— the same ruleCaseInsensitiveStringMapuses — so a scan-level override replaces the table-level option rather than sitting beside it. It is threaded through the read and write paths in place of the raw map, converting back withasMap()only where the native bindings are called.Two details worth flagging:
VortexBatchExec.createReaderFactory, so an invalid value fails the query once instead of once per task on the executors.rejectedWriteBatchSize()returns the key and the value that were ignored, so the warning names whichever key the user actually set (previously it could not distinguishbatch.sizefromvortex.write.batch.size, and dropped the offending value).Tests
New
VortexOptionsTest(19 cases): case-insensitive resolution for each option, the legacy batch-size key and its precedence, out-of-range fallback reported under the key the user set, validation messages, override semantics through Spark's own map,asMap()immutability, defensive copying, and a Java-serialization round trip that resolves an option before serializing so thetransientview is actually populated.VortexTableTestgains a case pinning the table-vs-scan option merge end to end.Each of these was checked for vacuity by mutating the implementation and confirming the test fails: dropping
transient, returning a new instance from thewithOverridesfast path, and makingnewScanBuilderignore its argument all produce failures../gradlew :vortex-spark_2.13:test— 176 tests pass./gradlew :vortex-spark_2.12:test— 176 tests passspotlessCheck(both Scala versions) andjavadoc— cleanVortexDataSourceS3MockTestfails locally for lack of a Docker environment, unrelated to this change.